Skip to content

Metrics files next to the model, and release 1.13.0 - #46

Merged
jarichb merged 6 commits into
mainfrom
claude/metrics-file
Oct 6, 2026
Merged

jarichb merged 6 commits into
mainfrom
claude/metrics-file

Conversation

@jarichb

@jarichb jarichb commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Adds an optional metrics file next to the model, <stem>.metrics.yml, with its own spec (metrics spec 0.1.0) and JSON Schema. It is the engine half of "metrics next to the model", decided on 6 Oct 2026; the studio builds its Metrics tab on top of it next. The full list is in the 1.13.0 CHANGELOG entry.

What it adds

  • The format. Four kinds of metric:

    • a simple aggregation of a column;
    • a row count;
    • a ratio of totals;
    • arithmetic over metric names (expression), the one documented exception to "nothing encoded in a string".

    Simple metrics and row counts take a where filter with eq, ne, in, not_in, gt, gte, lt, lte, between and is_null, combined with any and all, with nulls as in SQL. A filter may name a column on any table the metric's table reaches many-to-one, along exactly one path. Every measure column also yields an inferred metric, named as the studio's semantic export names them today.

  • model2data.metrics, the public API: load/validate, and resolve into one semantic representation (entities, relationships, dimensions, measures, metrics with their join paths). Also known_values, to_ossie and write_metric_tests.

  • Known values. Computed with DuckDB over the generated data, overall and per month, with the same SQL the dbt tests run. Counts are exact; everything else is rounded to 6 places, half to even.

  • CLI.

    • validate checks *.metrics.yml, against the model beside it by default.
    • generate --metrics FILE adds metric_values.json, one dbt test per metric in data-tests/metrics/, and an Ossie file in osi/.
    • metrics list and metrics export --to ossie work without generating.
  • Apache Ossie export, 0.1.1 as tagged osi-0.1.1-rc1. No final 0.1.1 tag exists yet. What Ossie has no field for goes into custom_extensions and is listed at the top of the file. Every example is checked against the vendored Ossie schema (Apache-2.0, with its LICENSE and NOTICE).

  • Docs: README, README_PYPI, LLMS.md, the guide text, the spec README, PUBLISHING.md rows for /spec/metrics/0.1.0/, and the CHANGELOG.

Determinism

  • The generator never reads the metrics file. No existing hash in determinism.json moved; two cases were added.
  • --metrics only adds files. test_metrics_only_add_files checks that every file shared with a run without it is byte-identical.

Decisions made while building

  • format: percent expects a fraction (0.6 shows as 60%), as BI tools do, and is never multiplied.
  • The metric tests live in data-tests/metrics/, because that is the generated project's test-paths.
  • generate does not pick up a metrics file beside the model on its own, so an unchanged command keeps writing the same files.
  • Order comparisons are refused on text columns too, because collation differs per warehouse.
  • Ossie datasets point at the engine's staging models (staging.stg_<table>).

Checked

  • Full suite: 2129 passed. ruff, ty and pre-commit are clean.
  • A real dbt build on four cases passes every metric test, including defects over several days. A known value moved past its tolerance makes its test fail.

Not verified

  • dbt Core 1.12 reading the osi/ folder, and Snowflake's OSI import.
  • The metric tests on Postgres. The Postgres CI job only seeds and runs.

After merge: publish the v1.13.0 release (PyPI), and host /model2data/spec/metrics/0.1.0/ on the website per PUBLISHING.md.

🤖 Generated with Claude Code

jarichb and others added 4 commits October 6, 2026 17:17
A metrics file, `<stem>.metrics.yml` beside the model, says what the
model's numbers mean: simple metrics (a column aggregated), row counts,
ratios of totals and derived metrics (arithmetic over metrics), with
filters that reach other tables through many-to-one foreign keys along
exactly one path, and nulls as in SQL. Every `measure` column of the model
also implies a metric, named as the studio's semantic export names it.

- model2data/spec/metrics/: the normative README and JSON Schema, versioned
  on its own (0.1.0), and a coffee_webshop example that conforms against the
  example model (which now marks total_amount and quantity as measures).
- model2data.metrics: the reader (the model spec's YAML profile, its Issue
  type and schema messages), the checks within the file and against the
  model, one resolved semantic representation (entities, relationships,
  dimensions, measures, metrics with their join paths and time), known values
  computed on DuckDB with the SQL the dbt tests run, a singular dbt test per
  metric, and an Apache Ossie 0.1.1 export with a lossiness report.

The generator never reads a metrics file.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…st|export

- `validate` checks a *.metrics.yml file against the metrics spec and its
  model: `--model`, else the model beside it with the same stem.
- `generate --metrics FILE` adds metric_values.json (each metric's known
  value over the seeds, overall and by month), a singular dbt test per metric
  under data-tests/metrics/ and osi/<project>.yml (Apache Ossie 0.1.1), and
  changes no other file. A sibling metrics file is not picked up on its own,
  so an unchanged command keeps writing the same files.
- `metrics list` and `metrics export --to ossie` work without generating; with
  no metrics file they use the model's inferred metrics.

The determinism suite gains two cases with metrics; every existing hash is
unchanged. The dbt cases run `dbt build` on the reference example, a model
with every kind of column and join, and the training model with defects over
several days, and require every metric test to pass.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
A "Metrics next to the model" section in both READMEs, with the coffee
example, what the known values and the metric tests are, and the Ossie
export; the metrics file and commands in LLMS.md and `guide tune`, and
what to do about their messages in `guide triage`. The guide's option
check now reads the options of grouped commands (`metrics export --to`).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Metrics next to the model: metrics spec 0.1.0, `validate` on metrics files,
`generate --metrics` (known values, a dbt test per metric, Apache Ossie
0.1.1), `metrics list|export`, and the `model2data.metrics` API.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@codecov

codecov Bot commented Oct 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

jarichb and others added 2 commits October 6, 2026 20:48
The README now leads with what model2data does today: one line per
capability with a link to its page, a before and after of a real run with
metrics (model and metrics in; the project, dbt build's summary and
metric_values.json out), who it is for, and how it works with the metrics
step in the diagram. README_PYPI.md carries the same top half with
absolute links and a text diagram.

The reference moved to docs/, one page per topic, worded as it was. Two
fixes on the way: the coffee_webshop metrics command now passes --rows 200
(the values quoted are for 200 rows), and the rounding line says what
metric_values.json says (integer sums, minimums and maximums are exact).

examples/ecommerce.metrics.yml gives the e-commerce example revenue,
orders, average order value and units sold, and the demo GIF shows them:
generate with --metrics, dbt build with the four metric tests passing, the
known values.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@jarichb
jarichb merged commit c8473a9 into main Oct 6, 2026
11 checks passed
@jarichb
jarichb deleted the claude/metrics-file branch October 6, 2026 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant